Skip to content

feat(core): Support schema management over REST in RESTCatalog - #9673

Merged
JingsongLi merged 3 commits into
apache:masterfrom
baiyangtx:zhangyongxiang.alpha/rest-catalog-schema-api
Sep 8, 2026
Merged

feat(core): Support schema management over REST in RESTCatalog#9673
JingsongLi merged 3 commits into
apache:masterfrom
baiyangtx:zhangyongxiang.alpha/rest-catalog-schema-api

Conversation

@baiyangtx

@baiyangtx baiyangtx commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Purpose

Follow up on #9174 by adding display-only schema history APIs to RESTCatalog.

The APIs follow the existing snapshot API pattern and allow clients to inspect complete historical TableSchema metadata through the REST service. Existing table operations and schema-management internals remain unchanged.

API

Catalog

Optional<TableSchema> loadSchema(Identifier identifier, String version);

PagedList<TableSchema> listSchemasPaged(
        Identifier identifier, Integer maxResults, String pageToken);

loadSchema supports:

  • LATEST
  • EARLIEST
  • a schema ID

listSchemasPaged returns schemas in descending schema ID order.

REST protocol

GET /v1/{prefix}/databases/{database}/tables/{table}/schemas/{version}
GET /v1/{prefix}/databases/{database}/tables/{table}/schemas?maxResults=<int>&pageToken=<token>

The responses contain complete TableSchema objects so that schema metadata is preserved.

Backward compatibility

  • The new endpoints are used only for schema display.
  • Existing getTable, alterTable, and rollbackSchema behavior continues to use the existing implementation.
  • No schema capability flag, catalog-backed SchemaManager, or CatalogEnvironment wiring is introduced.
  • New clients remain compatible with older REST servers that do not implement the schema endpoints.

Tests

Added coverage for:

  • loading LATEST, EARLIEST, and a specific schema ID
  • missing and invalid schema versions
  • paginated schema listing and ordering
  • missing-table error mapping
  • compatibility with a server that does not provide the new endpoints, including verification that existing get/alter/rollback operations never call them

Validation performed:

mvn spotless:apply
mvn -pl paimon-api,paimon-core -DskipTests compile
mvn -pl paimon-core -am -Pfast-build -DwildcardSuites=none \
  -DfailIfNoTests=false \
  '-Dtest=MockRESTCatalogTest#testLoadSchema+testListSchemasPaged+testSchemaMethodsTableNotExist+testCompatibilityWithServerWithoutSchemaEndpoints' \
  test

All targeted tests passed: 4 tests, 0 failures, 0 errors.

zhangyongxiang.alpha and others added 2 commits September 7, 2026 15:51
Add a Catalog-level API for listing schemas and expose it over the REST
protocol so that RESTCatalog-backed tables can read historical schemas
without direct filesystem access.

* Introduce `Catalog#supportsSchemaManagement` and
  `Catalog#listSchemas(Identifier, SchemaFilter)` with a
  `SchemaFilter` value object (all / latest / earliest / by id / by
  range).
* Add `GET /v1/{prefix}/databases/{db}/tables/{obj}/schemas` with a
  `ListSchemaResponse` payload; encode `SchemaFilter` as query
  parameters (`latest`, `earliest`, `schemaId`, `maxSchemaId`,
  `minSchemaId`).
* Implement `RESTCatalog#supportsSchemaManagement`/`listSchemas` and
  map REST errors to catalog exceptions.
* Add `CatalogSchemaManager`, a `SchemaManager` that delegates to the
  owning `Catalog` (analogous to `CatalogBranchManager`); reads go
  through `listSchemas`, writes reuse existing
  `Catalog#createTable`/`alterTable`/`rollbackSchema`.
* Wire `AbstractFileStoreTable#schemaManager` to prefer
  `CatalogSchemaManager` whenever `supportsSchemaManagement()` is true.
* Extend the REST mock server and add tests covering the new filter
  variants and the catalog-backed schema manager.

Co-authored-by: TRAE CLI <traecli@bytedance.com>
@baiyangtx

Copy link
Copy Markdown
Contributor Author

Hi @JingsongLi , this PR adds schema management support to RESTCatalog. Could you please take a look when you have time? Thanks!

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. API design

Could we follow the existing loadSnapshot and listSnapshotsPaged pattern here, with loadSchema(identifier, version) and listSchemasPaged(identifier, maxResults, pageToken)? The version could support LATEST, EARLIEST, and a schema ID. This would keep the API consistent and avoid introducing SchemaFilter. The responses should carry the complete TableSchema to preserve its metadata.

  1. Backward compatibility

These new APIs should only serve schema display use cases. Existing functionality must continue using its current implementation so that newer clients remain compatible with older REST servers.

Could we remove CatalogSchemaManager and the related capability flag and table-environment wiring from this PR? Please also add a compatibility test against a server without the new endpoints, verifying that existing operations still work and never call those endpoints.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit 7d7a672 into apache:master Sep 8, 2026
13 of 14 checks passed
@baiyangtx
baiyangtx deleted the zhangyongxiang.alpha/rest-catalog-schema-api branch September 9, 2026 13:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants